`format` as a annotation by default - separate vocabs (option 3) #1027

gregsdennis · 2020-11-21T20:49:29Z

There is also this in the "Custom format attributes" section below the edits:

An implementation MUST NOT fail validation or cease processing due to an unknown format attribute.

What is an implementation to do when an unknown format is encountered other than fail?

The context may be around collecting annotations, but it says "validation," so I'm confused. Maybe this needs to be clarified a bit too.

Resolves #1020.
Closes #1023.
Closes #1026.

Relequestual · 2020-11-21T22:24:42Z

There is also this in the "Custom format attributes" section below the edits:

An implementation MUST NOT fail validation or cease processing due to an unknown format attribute.

What is an implementation to do when an unknown format is encountered other than fail?

The context may be around collecting annotations, but it says "validation," so I'm confused. Maybe this needs to be clarified a bit too.

For annotation vocab, I would say, collect it as an annotation as long as it's a string. If it's not a string, then fail.

For assertion vocab, FAIL. The set of allowed values is explicit.

gregsdennis · 2020-11-21T22:33:36Z

I'll add this into the update.

(Validating that the format is a string is a meta-schema responsibility when validating the schema. That relationship always gets confusing.)

Relequestual

I'm not sure we were totally on the same page but close.
I was suggesting we should do away with saying anything about an implementation option to change annotation format to an assertion. It's confusing and was meant to only be a stop-gap till we found a solution.

Either you get annotations (default), and an implementation MAY do post-processing with the annotation results, but there should be no impact on the JSON Schema validation results.

Or you get full semantic validation.

jsonschema-validation.xml

karenetheridge · 2020-11-22T20:19:51Z

For annotation vocab, I would say, collect it as an annotation as long as it's a string. If it's not a string, then fail.

This can only be done since we know that all existing formats are for instance data that are strings, and no other type. Are we expecting to add additional formats to this vocabulary in the future, or will those have to be added as separate vocabularies?

edit: I wrote this thinking that "it's a string" was referring to the data instance. If instead you meant that the value of the 'format' property itself in the schema was the "it", then yes -- the value of format must always be a string, even if it's unrecognized, because that's what the format metaschema says.

karenetheridge · 2020-11-22T20:23:59Z

Another thought -- the format-as-assertion vocabulary should have an enum that lists the allowable values of the format keyword. Any unrecognized value should be a failure, and the metaschema can be used to perform that check.

Also: if we go ahead with this PR, we'll need new meta/*.json files for use by the metaschema (and the $vocabulary entry in the main metaschema file will need updates too).

jsonschema-validation.xml

karenetheridge

I'm not sure of some of the specific wording here, but I lean towards favouring the approach in this PR vs. the other alternatives.

gregsdennis · 2020-11-22T20:34:20Z

the format-as-assertion vocabulary should have an enum that lists the allowable values of the format keyword

This would conflict with the section on custom keywords. We need to allow for support for these, and having an enum of "approved" formats in the meta-schema would prevent that. This test of "known" formats must be performed by the implementation. Shouldn't be too hard, though: "I don't have a handler for this format... FAIL!"

karenetheridge · 2020-11-22T21:07:02Z

This would conflict with the section on custom keywords.

I had gotten the impression that we didn't want to continue to support custom formats -- if a user wants a new format they should create a new vocabulary for it. It would definitely be in the spirit of "making format more sane and less hand-wavey".

gregsdennis · 2020-11-22T21:10:28Z

That's understandable, but custom formats aren't the problem that this PR is trying to resolve. It's easy enough to continue supporting them for now.

…for format

gregsdennis · 2020-11-23T11:29:20Z

As a side note, even though I split the meta-schemas, I haven't updated the draft: they're still 2019-09, but they're aligned with the others.

Relequestual

Last change, but noticed the title for the meta-schema was the same for both format related meta-schemas, and this could be confusing.

If you could apply the same suggestions to both, I'll then approve and merge this PR.

meta/format-annotation.json

jsonschema-validation.xml

Relequestual · 2020-11-24T12:33:38Z

I'm so annoying, but can you

Modify the section title from A Vocabulary for Semantic Content With "format"' to Vocabularies for...
Update the general purpose meta-schema to include the new vocabulary URI
Add something to the changelog about this

THEN, it will merge. Honest.

Relequestual · 2020-11-24T22:48:18Z

Agreed with Greg, I will do the work as required in the above comment after merging this PR.

Relequestual · 2020-11-24T23:44:42Z

I have done the three items as listed above.

karenetheridge · 2020-11-25T01:33:49Z

jsonschema-validation.xml

@@ -514,18 +514,12 @@
            <section title="Foreword">
                <t>
                    Structural validation alone may be insufficient to allow an application to correctly
-                    utilize certain values.  The "format" annotation keyword is defined to allow schema
+                    utilize certain values. The "format" annotation keyword is defined to allow schema


can we keep unrelated whitespace changes out of the PR? it adds confusion and often conflicts with other things in flight.

This was requested by @Relequestual in Slack.

karenetheridge · 2020-11-25T01:35:37Z

jsonschema-validation.xml

+                    &lt;https://json-schema.org/draft/2020-11/vocab/format-annotation&gt;. The current
+                    URI for the corresponding meta-schema is:
+                    <eref target="https://json-schema.org/draft/2020-11/meta/format-annotation"/>.
+                    Implementing support for this vocabulary is REQUIRED.


suggested change (using the proper IETF language):

Implementations MUST support this vocabulary.

REQUIRED is proper language. RFC 2119 section 1.

karenetheridge · 2020-11-25T01:36:02Z

jsonschema-validation.xml

-                    &lt;https://json-schema.org/draft/2020-11/vocab/format&gt;.
+                    In addition to the Format-Annotation vocabulary, a secondary vocabulary is available
+                    for custom meta-schemas that defines "format" as an assertion. The URI for the
+                    Format-Assertion vocabulary, is:


remove comma

karenetheridge · 2020-11-25T01:36:34Z

jsonschema-validation.xml

+                    &lt;https://json-schema.org/draft/2020-11/vocab/format-assertion&gt;. The current
+                    URI for the corresponding meta-schema is:
+                    <eref target="https://json-schema.org/draft/2020-11/meta/format-assertion"/>.
+                    Implementing support for the Format-Assertion vocabulary is OPTIONAL.


suggested change:

Implementations MAY choose to support this vocabulary.

OPTIONAL is proper language. RFC 2119 section 5.

jsonschema-validation.xml

karenetheridge · 2020-11-25T01:39:13Z

jsonschema-validation.xml

-                        behavior described in the next section.
+                        Implementations MAY still treat "format" as an assertion in addition to an
+                        annotation and attempt to validate the value's conformance to the specified
+                        semantics. The implementation MUST provide options to enable and disable such


Why MUST they, if it can also be done via $vocabulary? I would soften this to a SHOULD. Ideally, the fewer knobs needed in an implementation outside of specifying a $schema, the better.

This is solely for when the annotation vocabulary is used. If an implementation supports assertion (even partial assertion) they MAY choose to treat the keyword as such. If they do, they MUST allow this feature to be enabled and disabled.

The assertion vocabulary doesn't have this allowance, instead specifying that implementations MUST implement assertion fully in order to support the vocabulary.

I'm still confused. It sounds like a schema (via the metaschema) can specify to use the format-annotation vocabulary, but it might still get assertion behaviour anyway (when using an implementation that supported validations)? That would force the setting of an out-of-band configuration "no, really, when I say annotations I mean only annotations" and there would be no other way to not get validation behaviour as well. If I'm correct so far, then the configuration value MUST default to false so validation behaviour doesn't happen accidentally.

The schema can say, "You're getting annotations."

But this bit of the spec allows the client/application to say, "I know I'm supposed to get annotations, but this library I'm using supports asserting formats, and that'd save me some work, so I'd like to use that." It gives the choice to the client.

It also the opens the door for implementations to provide partial validation, which is a pain point that @handrews had to deal with (pressure from implementers) and what the previous wording of this section tried so hard to allow.

However, if the schema says, "You're getting assertions," then the implementation MUST be able to provide validation or refuse to process the schema.

The last scenario is <format-assertion>: false. This means, "If you support asserting these things, that's great, please do. If not, it's cool; they'll be annotations (because you don't understand them)." But this is enforced by the schema, not opted-in by the application.

In the end, we have

Annotation
Vocab Assertion
Vocab Impl Assert
Support App Config Result

false/true n/a no n/a annotation

false/true n/a yes annotation annotation

false/true n/a yes assertion assertion

n/a false no n/a annotation

n/a false yes annotation/assertion assertion

n/a true no n/a fail

n/a true yes annotation/assertion assertion

But this bit of the spec allows the client/application to say, "I know I'm supposed to get annotations, but this library I'm using supports asserting formats, and that'd save me some work, so I'd like to use that." It gives the choice to the client.

My main point here is that if the implementation can do this, the setting MUST default to NOT doing it. Users must explicitly opt in. Otherwise, there is no way of a schema being processed as annotation only (with any combination of $schema and $vocabulary settings) with an implementation that can also handle assertions.

the setting MUST default to NOT doing it

Absolutely. That is covered on the next line: "... evaluation and MUST be disabled by default."

karenetheridge · 2020-11-25T01:40:00Z

jsonschema-validation.xml

+                    <t>
+                        When the Format-Assertion vocabulary is declared with a value of true,
+                        implementations MUST provide full validation support for all of the formats
+                        defined by this specificaion. Implementations that cannot provide full


typo "specificaion"

karenetheridge · 2020-11-25T01:44:38Z

whoops, looks like I hit submit a bit too late...

Relequestual · 2020-11-25T09:31:04Z

Ah nuts sorry @karenetheridge - My internet was super flakey yesterday (and few days before that, and still is).
There are a few obvious fixes needed here. Feel free to open a new PR, or I'll try to cover all your comments in final edits on Friday. =] Many thanks

issue-1020-separate-format-vocabularies

62b9c21

gregsdennis requested a review from Relequestual November 21, 2020 20:50

add clarification around custom formats

9949508

gregsdennis added this to the draft-08-patch1 (draft 2020-NN) milestone Nov 21, 2020

Relequestual requested changes Nov 21, 2020

View reviewed changes

jsonschema-validation.xml Outdated Show resolved Hide resolved

jsonschema-validation.xml Show resolved Hide resolved

This was referenced Nov 21, 2020

format is an annotation by default (option 2) #1026

Closed

reworked 'format' to default to annotation except when configured for assertion #1023

Closed

fixed 'assertation' typo

9d29a9f

karenetheridge reviewed Nov 22, 2020

View reviewed changes

jsonschema-validation.xml Outdated Show resolved Hide resolved

karenetheridge reviewed Nov 22, 2020

View reviewed changes

jsonschema-validation.xml Outdated Show resolved Hide resolved

karenetheridge reviewed Nov 22, 2020

View reviewed changes

split format meta-schemas; explicitly specify implementation support …

8acb128

…for format

gregsdennis requested a review from Relequestual November 23, 2020 05:47

Relequestual requested changes Nov 23, 2020

View reviewed changes

meta/format-annotation.json Outdated Show resolved Hide resolved

Relequestual requested changes Nov 23, 2020

View reviewed changes

jsonschema-validation.xml Outdated Show resolved Hide resolved

gregsdennis requested review from Relequestual and karenetheridge November 24, 2020 04:37

update title for meta-schemas

242cfd0

Relequestual approved these changes Nov 24, 2020

View reviewed changes

Relequestual added the format label Nov 24, 2020

Relequestual merged commit 2ce4968 into master Nov 24, 2020

Relequestual deleted the separate-format-vocabularies branch November 24, 2020 23:44

karenetheridge reviewed Nov 25, 2020

View reviewed changes

gregsdennis mentioned this pull request Jul 5, 2021

Tests for format should make better use of vocabularies and optional/ is too overloaded with meaning json-schema-org/JSON-Schema-Test-Suite#495

Open

Relequestual mentioned this pull request Mar 28, 2023

Incorrect Revert "by default, "format" only annotates, not validates"? json-schema-org/JSON-Schema-Test-Suite#660

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`format` as a annotation by default - separate vocabs (option 3) #1027

`format` as a annotation by default - separate vocabs (option 3) #1027

gregsdennis commented Nov 21, 2020 •

edited

Loading

Relequestual commented Nov 21, 2020

gregsdennis commented Nov 21, 2020 •

edited

Loading

Relequestual left a comment

karenetheridge commented Nov 22, 2020 •

edited

Loading

karenetheridge commented Nov 22, 2020 •

edited

Loading

karenetheridge left a comment

gregsdennis commented Nov 22, 2020 •

edited

Loading

karenetheridge commented Nov 22, 2020

gregsdennis commented Nov 22, 2020

gregsdennis commented Nov 23, 2020

Relequestual left a comment

Relequestual commented Nov 24, 2020

Relequestual commented Nov 24, 2020

Relequestual commented Nov 24, 2020

karenetheridge Nov 25, 2020

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020

karenetheridge Nov 25, 2020

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020 •

edited

Loading

gregsdennis Nov 25, 2020 •

edited

Loading

gregsdennis Nov 25, 2020 •

edited

Loading

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020

gregsdennis Nov 25, 2020

karenetheridge Nov 25, 2020

karenetheridge commented Nov 25, 2020

Relequestual commented Nov 25, 2020

Annotation Vocab	Assertion Vocab	Impl Assert Support	App Config	Result
false/true	n/a	no	n/a	annotation
false/true	n/a	yes	annotation	annotation
false/true	n/a	yes	assertion	assertion
n/a	false	no	n/a	annotation
n/a	false	yes	annotation/assertion	assertion
n/a	true	no	n/a	fail
n/a	true	yes	annotation/assertion	assertion

format as a annotation by default - separate vocabs (option 3) #1027

format as a annotation by default - separate vocabs (option 3) #1027

Conversation

gregsdennis commented Nov 21, 2020 • edited Loading

Relequestual commented Nov 21, 2020

gregsdennis commented Nov 21, 2020 • edited Loading

Relequestual left a comment

Choose a reason for hiding this comment

karenetheridge commented Nov 22, 2020 • edited Loading

karenetheridge commented Nov 22, 2020 • edited Loading

karenetheridge left a comment

Choose a reason for hiding this comment

gregsdennis commented Nov 22, 2020 • edited Loading

karenetheridge commented Nov 22, 2020

gregsdennis commented Nov 22, 2020

gregsdennis commented Nov 23, 2020

Relequestual left a comment

Choose a reason for hiding this comment

Relequestual commented Nov 24, 2020

Relequestual commented Nov 24, 2020

Relequestual commented Nov 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenetheridge Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

gregsdennis Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

gregsdennis Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenetheridge commented Nov 25, 2020

Relequestual commented Nov 25, 2020

`format` as a annotation by default - separate vocabs (option 3) #1027

`format` as a annotation by default - separate vocabs (option 3) #1027

gregsdennis commented Nov 21, 2020 •

edited

Loading

gregsdennis commented Nov 21, 2020 •

edited

Loading

karenetheridge commented Nov 22, 2020 •

edited

Loading

karenetheridge commented Nov 22, 2020 •

edited

Loading

gregsdennis commented Nov 22, 2020 •

edited

Loading

karenetheridge Nov 25, 2020 •

edited

Loading

gregsdennis Nov 25, 2020 •

edited

Loading

gregsdennis Nov 25, 2020 •

edited

Loading